gfx1250 opus gemm splitk fuse - #4246

Add the fused split-k pipeline and split-k reduce launch for gfx1250 opus
gemm, extend split-k reduce to support the bf16 workspace + fp32-bias path,
and update codegen/gen_instances to emit the fused instances.

Co-authored-by: Cursor <cursoragent@cursor.com>

The aiter imports must run after _patch_flaky_hip_device_count(); mark them
with noqa: E402 to satisfy the ruff code-style CI.

Co-authored-by: Cursor <cursoragent@cursor.com>

…sults

Overwrite the 59 tuned gfx1250 K=7168 shapes with the latest tuning run;
some large-M shapes now select triton over opus.

Co-authored-by: Cursor <cursoragent@cursor.com>

Add a golden-checked CUDA-graph opus_gemm sweep (default entry) with split-K
workspace pre-warm on the capture stream, and print a latency-vs-tuned-CSV
table for gfx1250.

Co-authored-by: Cursor <cursoragent@cursor.com>

…1250 dispatch

- Add opus_gemm_workspace_release() (current stream) and
  opus_gemm_workspace_release_all() (all streams) to free the per-stream
  split-K workspace buffer, host/device handles and registry entry, with
  eager-mode guards and pre-free synchronization; wire up pybind + Python.
- Make the gfx1250 dispatch self-contained: move the shared flat-array
  dispatch POD types into opus_gemm_heuristic_dispatch_gfx1250.cuh
  (opus_gfx1250_detail) and drop the gfx950 header include so a gfx1250-only
  build compiles without OPUS_BUILD_HAS_GFX950.
- Allocate the split-K workspace handle from plain host memory on the
  gfx942/gfx1250 (device-mirror) path instead of hipHostMallocCoherent, so
  those builds use no pinned/coherent host memory (reclaimed by the OS at
  process exit); gfx950 keeps the coherent handle it dereferences on device.

Co-authored-by: Cursor <cursoragent@cursor.com>

…try)

Replace the SplitkWsRegistry/hipMalloc-based workspace management for gfx1250
split-K kernels with direct pointer passing from Python-allocated torch.empty
buffers (bf16 by default, matching OPUS_WS_BF16=1).

Changes:
- kargs struct: ws_handle* -> void* ptr_ws (direct pointer, no indirection)
- main kernels: kargs.ws_handle->ptr -> kargs.ptr_ws
- reduce kernel: const opus_splitk_ws_handle* -> const void* ws_ptr
- codegen launcher: remove registry/grow logic, add workspace tensor param
- C++ dispatch: opus_gemm_a16w16_tune gets optional workspace param;
  gfx1250 split-K kids pass workspace, others unchanged
- Python layer: allocate workspace via torch.empty(dtype=bf16) for kid [20000,30000)
- OpusA16W16NoscaleKernel for gfx1250 moved to opus_gfx1250_detail namespace
  (6-arg signature with workspace, separate from gfx950/gfx942 5-arg type)
- gen_instances.py manifest: emit 6-arg forward decl for gfx1250 split-K launchers
- gfx950/gfx942 paths unchanged (still use SplitkWsRegistry)

Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: binglue.lyu <binglue.lyu@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

…rewarm

The gfx1250 split-K partial-sum workspace was allocated as a bf16 buffer sized
from an over-estimated raw element count. Kids whose kernel writes an fp32
workspace then overran the buffer (fp32 is 2x the bytes), corrupting adjacent
device memory and hanging the machine.

Size the workspace from the selected kid's OWN kernel definition instead:
  * allocate a [batch, split_k, padded_M, padded_N] tensor by element count in
    the kid's workspace dtype (fuse kids use compile-time fuse_split_k /
    fuse_ws_dtype; ws-variant kids use fp32 with runtime split_k), never a raw
    byte size;
  * fall back to a safe fp32 over-estimate when the kid table can't be loaded.

Also drop the eager graph-capture prewarm in tuned_gemm: the workspace is a
plain torch.empty passed to the launcher, and torch's HIP graph-capture-aware
caching allocator makes a single torch.empty path valid for both eager and
capture (replay-stable address, no pre-warm needed). The gfx942/gfx950 hipMalloc
registry path is unaffected and still fails loudly if used under cudagraphs
without a warm.

Co-authored-by: Cursor <cursoragent@cursor.com>

Replaces the positional tdm_desc constructor with tdm_traits::resolve<> over an
unordered pack of policy tags, so a call site names only the concerns it varies
and every default and validation lives in one place.

Adds the dim1..dim3 stride programming that 3D-5D windows were missing, so a
tile past 2D walks planes by the D#'s own pitch instead of landing on top of
itself, and rejects gather with ndim>2 at compile time since both claim D#
groups 2/3.

Moves the LDS write point out of window state into a make_descriptor()
parameter, leaving slot cycling to the pipeline that already owns the
s_wait_tensorcnt depth.

Makes every offset the public API takes an element count, matching the extents,
strides and origins it already took. The LDS byte scaling folds into the
address add, so the 4wave GEMM comes out 10 instructions shorter.

…g 24

The two gather bits in D# group 0 dword 0 were transposed. The reference
lowering in llvm-project (AMDGPUMakeDmaBaseLowering in AMDGPUToROCDL.cpp, with
the offsets pinned by name in the gfx1250 conversion test) puts gather_mode at
bit 30 and index_size at bit 31; opus had them the other way round. gather<32>
sets both bits either way, which is why the encoding was accidentally correct
everywhere it had been exercised, but gather<16> emitted 0x80000001 where the
reference emits 0x40000001 -- index_size of 32 bits with gather disabled, so the
window silently degraded to a plain contiguous copy instead of failing. Since
async_store shares make_descriptor(), scatter carried the same bug and is fixed
with it. The rest of the descriptor was checked field by field against the same
lowering and agrees, as does the 16-bit index packing and the reuse of tile_dim1
as the valid-index count.

Also drops the runtime rewrite of those bits in make_descriptor(). Both are
compile-time and already sit in group0_dword0_const, which nothing else patches,
so it was dead -- and holding one encoding in two places is what let the
transposition survive.

Three clang-24 builtin changes broke the header outright. aac296b87e8c moved the
AMDGPU builtins from GNU vectors to ExtVector, so the lax conversions that used
to paper over spelling differences are gone, and 7318bc734a5b retyped D# group 0
as unsigned while leaving groups 1-3 signed. fp16_t is therefore registered as
_Float16 from clang 24 on, which is what the fp16 wmma/mfma builtins now demand
and what opus itself used before clang 20; the older compilers keep __fp16
rather than shifting their numerics, since respelling turns arithmetic on fp16_t
from the fp32-intermediate v_fma_mixlo_f16 into a native v_fma_f16. Group 0 gets
a bit_cast behind OPUS_HAS_UNSIGNED_TENSOR_D0, generating nothing since the
lanes are unchanged, and the b64/b128 buffer store vdata goes unconditionally
unsigned because it was declared that way in the .def era too. Verified to
produce identical instructions under clang 23 and clang 24.

Tightens the sub-byte element check. Only array<fp4_t, 2> and its vector form
are accepted now: the bare type reports sizeof 1 for a 4-bit value so the
descriptor strides twice as far as the data, and an array or vector of any other
width makes the D# element several values at once, rescaling every extent,
stride and pad. The old sizeof test caught only the bare type.

The remainder is presentation. tdm_traits becomes a namespace rather than a
struct of statics, tdm_desc reaches through one Traits alias instead of five,
default_cache_policy comes from make_cache_policy() now that it is not stuck in
an incomplete-class context, and the longer comment blocks are condensed to the
file's single-line style with the measurements kept.

Co-authored-by: Cursor <cursoragent@cursor.com>

The gather mode and index size bits go back where they were. The previous
commit moved gather_mode to bit 30 and index_size to 31 to agree with MLIR's
AMDGPUToROCDL lowering; on gfx1250 the lowering is the one that is wrong.
Issuing one descriptor twice with nothing but that dword changed, 0x80000001
fetches the listed rows and 0x40000001 returns consecutive rows from the
origin. Only gather<16> can tell the two orderings apart -- gather<32> is
0xC0000001 either way -- which is how the transposition passed review and why
no 32-bit workload noticed it.

tdm_groups23 now receives tile_dim3 and tile_dim4 as template arguments.
tdm_desc computed them into group2_dword3_const and group3_dword2_const, but
the groups23 aggregate zeroed both operand groups and neither constant was
ever read, so a rank-4 tile collapsed onto its dim3 = 0 slice and a rank-5
tile onto dim3 = dim4 = 0. Ranks 2 and 3 were unaffected because tile_dim2
rides in group1, which is why nothing before this noticed.

test_tdm_feature_gfx1250.cu is the suite that found both: one self-contained
TU covering element sizes 1/2/4/8 bytes and array<fp4_t,2>, tile ranks 2..5
each with a move along its outermost dimension, gather and scatter at both
index widths, runtime descriptor state, padding, multicast and clamping.
Every check is byte-exact against a position hash, and untouched memory is
checked as hard as copied memory -- clamping, short gather lists and the
tile_dim1 patch are only visible that way. Both halves of the group0 encoding
are asserted, so a future transposition goes red instead of silently costing
gather<16> its rows.

test_tdm_gfx1250.cu is removed: it is written against tdm_window, which the
refactor took out of opus.hpp, so it has not compiled for some time. Its
coverage -- a pipelined multi-wave GEMM with cluster multicast -- is not what
the new suite checks and wants porting rather than deleting.

Around the harness: setup.py gains a positive per-arch list so the suite is
built only for gfx1250, an arch fallback through torch for hosts without
rocminfo, and named entries for two sources that do not currently compile
there, so one stale source no longer takes the whole .so with it. The
wmma_scale tests gain the missing-symbol guard the rest of the harness
already uses, since they look their launchers up by name at call time.

Verified on gfx1250 under both the ROCm 7.x clang and the clang 24 / HIP 7.15
toolchain in the opus_rocm_compiler container: 26 runtime checks pass, the 17
compile-time rejections all fire, opus_tdm_example is unaffected, and the
bf16 GEMM still runs at 147 TFlops.

The "feature" qualifier distinguished nothing -- it is the only TDM suite in
the directory. setup.py's source list and arch-only map follow the new name.

Signed-off-by: binglue.lyu <binglue.lyu@amd.com>

The TDM refactor replaced the positional tdm_window<> constructor with an
unordered pack of policy tags and moved every offset to element units, which
left the three gfx1250 a16w16 split-K pipelines and their traits header on an
API that no longer exists.

Rebuild WindowA/WindowB on opus::tdm<> with a padding_auto tag in place of the
kLdsPadEn / kPadInterval / kPadAmount constants the old descriptor carried, and
retire the byte-denominated slot sizes in favour of kSlotElemsA/B. The producer
loops now take the slot index as a template parameter and hand
make_descriptor() an element offset, so the double-buffer half is chosen at the
issue site instead of living in window state. The clusterlaunch variants read
their multicast masks from tdm_traits::peers_along_y/x<> rather than deriving
them in a loop, and the fuse variant's workspace staging follows the same
element-offset form.

Retune the gfx1250 blocks of the dsv4 and kimik3 bf16 GEMM configs (180 + 345
shapes) against opus and triton measured back to back in one session, keeping
the faster backend per shape: 212 shapes select opus and none regress against
the triton fallback they replace. The sweep searched the 496 plain +
clusterlaunch kids; OPUS_TUNE_NO_FUSE=1 is the switch that held the ~1.4k fuse
kids out of the compile set, so those remain unmeasured.

Co-authored-by: Cursor <cursoragent@cursor.com>

tdm_window<> and its positional make() no longer exist. The 2D x-load becomes
an opus::tdm<> window whose double-buffer half is handed to async_load() as an
element count, rather than being folded into the LDS base address in bytes.

The 3D residual load has no make() overload to move to, so it programs the
descriptor through make_from_layout() with explicit shape, pitch and origin
arrays in D# order. That also retires the placeholder barrier argument the old
eight-parameter call had to thread through.

Co-authored-by: Cursor <cursoragent@cursor.com>

The style CI has been red on this branch. Three separate findings, all on lines
the branch adds:

ruff flags the five `# noqa: E402` directives on the imports that have to run
after _patch_flaky_hip_device_count(). Under the 0.16.0 the workflow pins,
those imports no longer trip E402 at all, so the suppressions are dead and
RUF100 reports them. Drop them; the comment above the block still records why
the imports sit where they do.

ruff also flags two idioms in gemm_op_a16w16.py that reviewdog already
commented on: lru_cache(maxsize=None) becomes functools.cache, and the
splitK>1 ternary becomes max(1, ...).

black wants the _alloc_splitk_workspace call collapsed onto one line, which it
now fits after those edits.

Co-authored-by: Cursor <cursoragent@cursor.com>

Conflicts came from main's new a8w8_mxscale BMM family (#4320) landing in the
same two files this branch edits, and both sides were purely additive:

* opus_gemm_common.py OpusGemmInstance: keep this branch's fuse_* axes next to
  main's mxscale BMM axes; both groups are defaulted trailing fields.
* gen_instances.py manifest emit: keep both signature templates and fold the
  gfx1250 6-arg workspace choice into main's A16W16_TUNE_TAGS branch, after the
  BMM branch (the tag sets are disjoint, so the order is free).

Verified after the merge: 496 gfx1250 kids with none in the fuse band, tuner
candidate counts unchanged, and codegen over gfx950+gfx942+gfx1250 emits both
the 44 BMM and the 63 workspace launcher declarations.

Co-authored-by: Cursor <cursoragent@cursor.com>

…mpile

opus_gemm_lookup.h and opus_gemm_a16w16_tune_lookup.h each defined one
macro per output dtype, listing every kid in the compile subset regardless
of arch. Each arch's dispatch header then expanded those macros into a
table of its own launcher function-pointer type -- but gfx1250's a16w16
launchers take a workspace tensor that gfx950/gfx942's do not, so any build
targeting gfx1250 alongside another arch failed to type-check: gfx950's
table held gfx1250 function pointers and vice versa. GPU_ARCHS filtering
of |S| hid this for single-arch builds only. A gfx950+gfx1250 host compile
produced 439 such errors across all six tables.

Emit one macro per (dtype, arch) instead and have each arch header expand
only its own, which is also what the per-arch host TU split already does
for the split-K reduce ABI. Table contents are unchanged for every
single-arch build -- verified entry-for-entry against the base commit for
gfx950, gfx942 and gfx1250, with the rest of the generated blob byte
identical.

Winners in get_tune_dict are now keyed on arch too. The (M,N,K) tables are
per arch, so without arch in the key a shape tuned on two arches kept only
whichever CSV row was read last and the other arch silently fell back to
its heuristic: an all-arch build dropped 7 of 308 tuned rows (gfx1250
266/268, gfx950 35/40) before this change and drops none after.

Verified by host-compiling opus_gemm.cu for gfx950;gfx1250,
gfx942;gfx950;gfx1250, gfx942;gfx1250 and each arch alone.

Note that a full mixed-arch device build is still blocked by an unrelated
issue: the gfx1250 clusterlaunch pipeline applies __cluster_dims__ to its
kernel unguarded, which the gfx950 device pass rejects. That needs an
arch-guarded device stub and is not addressed here.

Co-authored-by: Cursor <cursoragent@cursor.com>

…the cluster kernels

Both clusterlaunch kernels already compile their body away to an empty stub
outside gfx1250, but the __cluster_dims__ attribute sat outside that guard.
A mixed-arch build compiles every device TU once per offload arch, and the
gfx950/gfx942 pass rejects the attribute before it ever reaches the body:

  opus_gemm_pipeline_a16w16_clusterlaunch_tdm_splitk_ws_gfx1250.cuh:75:1:
  error: 'cluster_dims' is not supported for this GPU architecture
  1 warning and 1 error generated when compiling for gfx950.

Guard it the same way, keeping the attribute on the host pass, which is
where the launch reads the cluster geometry from. Extended to the fuse
kernel too -- that family is unregistered today, so it has no TU to fail in,
but the bug is identical and would come back with it.

gfx1250 codegen is unchanged by construction (__gfx1250__ is defined on that
pass). Verified: gfx1250 device TUs now compile clean with
--offload-arch=gfx1250 --offload-arch=gfx950, a gfx1250-only build still
produces a correct GEMM (rel_err 2.7e-03 bf16), and four compiled
clusterlaunch kids dispatched by id are numerically clean (rel_err 5.4e-05),
so cluster multicast is intact.

A full mixed-arch build now gets past the gfx1250 TUs and hits the mirror
image of this problem, which is not addressed here: gfx950's kid TUs are
also compiled for gfx1250, where get_warp_size() is 32 instead of 64, so
wave-size assertions in the gfx950 traits fail (117 errors, e.g. "splitk
BLOCK_SIZE must cover exactly 4 waves"). Those TUs have no business being
compiled for another arch at all -- the fix belongs in the build flags
(per-source --no-offload-arch), not in more source guards.

Co-authored-by: Cursor <cursoragent@cursor.com>

…s compile

hipcc runs every TU once per --offload-arch, so a GPU_ARCHS=gfx950;gfx1250
build also instantiated the gfx950 kids for gfx1250: the gfx950 traits then
derived their layouts from a 32-wide wave and tripped
`BLOCK_SIZE == 4 * get_warp_size()`, and the reverse direction reached
gfx1250-only kernel attributes.

Wrap the generated device TUs (per-kid instances and the per-arch splitk
reduce) in `#if !defined(__HIP_DEVICE_COMPILE__) || defined(__<arch>__)`,
covering the #include as well, so a foreign device pass is an empty TU that
never parses another arch's headers. The host pass stays inside the guard
since it emits the __device_stub__ symbols the fused host TU links against.

Each object now carries its kernel only in its own arch's code object
(verified with llvm-nm on the unbundled images for gfx942/gfx950/gfx1250).
Builds clean for gfx950, gfx942, gfx1250, gfx950;gfx1250 and all three at
once; op_tests/test_opus_a16w16_gemm.py passes 15/15 on gfx1250 against both
the gfx1250-only and the mixed module.

Co-authored-by: Cursor <cursoragent@cursor.com>

…gemm-splitk-fuse

The GLM 5.x tuning commit added a block_m grid divisor but only assigned it
in the gfx950 branch, so gfx1250/gfx942 raised UnboundLocalError at launch.
Those kernels take no BLOCK_M and run one query row per program, so 1
restores the original (seq_len,) grid.

Co-authored-by: Cursor <cursoragent@cursor.com>